Method and system for image based occupancy detection

ABSTRACT

An image sensor includes an active pixel array including a number of pixels and image sensor control circuitry configured to perform a read operation only on a subset of the pixels of the active pixel array such that pixels not in the subset remain inactive. By reading out only the subset of pixels in the active pixel array and keeping the remaining pixels inactive, the temperature of the active pixel array may be reduced compared to a conventional read out process, thereby reducing thermal noise in the resulting pixel data.

FIELD OF THE DISCLOSURE

The present disclosure relates to methods and systems for detectingoccupancy using images. In particular, the present disclosure relates tomethods and systems for detecting occupancy that increase the efficiencyand accuracy of occupancy detection via an image sensor.

BACKGROUND

Modern lighting fixtures often include additional features above andbeyond their ability to provide light. For example, many lightingfixtures now include communications circuitry for sending and receivingcommands to and from other devices, control circuitry for setting thelight output thereof, and sensor circuitry for measuring one or moreenvironmental parameters. Recently, lighting fixtures have begun toincorporate image sensors. Image sensors in lighting fixtures aregenerally expected to detect occupancy (i.e., the presence of a person)in the area within the field of view of the image sensor. While thereare several well-known methods for determining occupancy using an imagesensor, these methods are complex and computationally expensive. As aresult, lighting fixtures utilizing an image sensor to detect occupancymust include relatively powerful processing circuitry, which consumesadditional power and drives up the cost of the lighting fixture.Accordingly, there is a need for systems and methods for detectingoccupancy using an image sensor with reduced complexity andcomputational expense.

SUMMARY

In one embodiment, an image sensor includes an active pixel arrayincluding a number of pixels and image sensor control circuitryconfigured to perform a read operation only on a subset of the pixels ofthe active pixel array such that pixels not in the subset remaininactive. By reading out only the subset of pixels in the active pixelarray and keeping the remaining pixels inactive, the temperature of theactive pixel array may be reduced compared to a conventional read outprocess, thereby reducing thermal noise in the resulting pixel data.

In one embodiment, a method for detecting occupancy from an image sensorincludes obtaining pixel data from the image sensor and analyzing thepixel data to determine if a person has entered the field of view of theimage sensor. Notably, the pixel data includes pixel values only for asubset of pixels in an active pixel array of the image sensor. Byobtaining and analyzing pixel data only for a subset of pixels, thecomputational expense of determining if a person has entered the fieldof view of the image sensor may be significantly reduced.

Those skilled in the art will appreciate the scope of the presentdisclosure and realize additional aspects thereof after reading thefollowing detailed description of the preferred embodiments inassociation with the accompanying drawing figures.

BRIEF DESCRIPTION OF THE DRAWING FIGURES

The accompanying drawing figures incorporated in and forming a part ofthis specification illustrate several aspects of the disclosure, andtogether with the description serve to explain the principles of thedisclosure.

FIG. 1 illustrates an image sensor according to one embodiment of thepresent disclosure.

FIG. 2 illustrates a pixel of an active pixel array according to oneembodiment of the present disclosure.

FIG. 3 is a flow diagram illustrating a method for detecting occupancyusing an image sensor according to one embodiment of the presentdisclosure.

FIG. 4 illustrates a read out pattern for an active pixel arrayaccording to one embodiment of the present disclosure.

FIG. 5 is a flow diagram illustrating a method for detecting occupancyusing an image sensor according to one embodiment of the presentdisclosure.

FIGS. 6A through 6C illustrate read out patterns for an active pixelarray according to various embodiments of the present disclosure.

FIG. 7 illustrates an intelligent lighting fixture according to oneembodiment of the present disclosure.

FIG. 8 illustrates an intelligent lighting network according to oneembodiment of the present disclosure.

FIG. 9 is a flow diagram illustrating operation of a lighting fixtureaccording to one embodiment.

FIG. 10 is a flow diagram illustrating occupant detection,classification, tracking, and handoff for a given lighting fixtureaccording to one embodiment.

FIGS. 11A through 11F illustrate movement of occupants through anexemplary field of view.

FIG. 12A illustrates fields of view prior to a mapping operation.

FIG. 12B illustrates fields of view after a mapping operation.

DETAILED DESCRIPTION

The embodiments set forth below represent the necessary information toenable those skilled in the art to practice the embodiments andillustrate the best mode of practicing the embodiments. Upon reading thefollowing description in light of the accompanying drawing figures,those skilled in the art will understand the concepts of the disclosureand will recognize applications of these concepts not particularlyaddressed herein. It should be understood that these concepts andapplications fall within the scope of the disclosure and theaccompanying claims.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another. For example, a first element could be termed asecond element, and, similarly, a second element could be termed a firstelement, without departing from the scope of the present disclosure. Asused herein, the term “and/or” includes any and all combinations of oneor more of the associated listed items.

It will be understood that when an element such as a layer, region, orsubstrate is referred to as being “on” or extending “onto” anotherelement, it can be directly on or extend directly onto the other elementor intervening elements may also be present. In contrast, when anelement is referred to as being “directly on” or extending “directlyonto” another element, there are no intervening elements present.Likewise, it will be understood that when an element such as a layer,region, or substrate is referred to as being “over” or extending “over”another element, it can be directly over or extend directly over theother element or intervening elements may also be present. In contrast,when an element is referred to as being “directly over” or extending“directly over” another element, there are no intervening elementspresent. It will also be understood that when an element is referred toas being “connected” or “coupled” to another element, it can be directlyconnected or coupled to the other element or intervening elements may bepresent. In contrast, when an element is referred to as being “directlyconnected” or “directly coupled” to another element, there are nointervening elements present.

Relative terms such as “below” or “above” or “upper” or “lower” or“horizontal” or “vertical” may be used herein to describe a relationshipof one element, layer, or region to another element, layer, or region asillustrated in the Figures. It will be understood that these terms andthose discussed above are intended to encompass different orientationsof the device in addition to the orientation depicted in the Figures.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the disclosure.As used herein, the singular forms “a,” “an,” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises,”“comprising,” “includes,” and/or “including” when used herein specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this disclosure belongs. It willbe further understood that terms used herein should be interpreted ashaving a meaning that is consistent with their meaning in the context ofthis specification and the relevant art and will not be interpreted inan idealized or overly formal sense unless expressly so defined herein.

FIG. 1 shows an image sensor 10 according to one embodiment of thepresent disclosure. The image sensor 10 includes an active pixel array12, control circuitry 14, a pixel selection circuitry 16, samplingcircuitry 18, analog-to-digital converter circuitry 20, an outputregister 22, and an output 24. The control circuitry 14 is coupled toeach one of the pixel selection circuitry 16, the sampling circuitry 18,the analog-to-digital converter circuitry 20, and the output register22. The pixel selection circuitry 16 is coupled to the active pixelarray 12. The sampling circuitry 18 is coupled between the active pixelarray 12 and the analog-to-digital converter circuitry 20. Theanalog-to-digital converter circuitry 20 is coupled to the outputregister 22, which is in turn coupled to the output 24.

In operation, the control circuitry 14 provides control signals to eachone of the pixel selection circuitry 16, the sampling circuitry 18, theanalog-to-digital circuitry 20, and the output register 22 to facilitatecapturing an image frame and providing a digitized version thereof atthe output 24 of the image sensor 10. The pixel selection circuitry 16selects one or more pixels in the active pixel array 12 to be resetand/or read out. In a conventional rolling shutter read process, thepixel selection circuitry 16 serially selects rows of pixels in theactive pixel array 12 to be reset and subsequently read out one afterthe other. Selected pixels provide analog signals proportional to anamount of light detected thereby to the sampling circuitry 18. Theanalog-to-digital converter circuitry 20 digitizes the analog signalsfrom the sampling circuitry 18 into pixel data and provides the pixeldata to the output register 22, where it can be retrieved via the output24.

FIG. 2 shows details of a pixel 26 in the active pixel array 12according to one embodiment of the present disclosure. The pixel 26includes a light detecting element 28 and support circuitry 30. Thelight detecting element 28 may be a photodiode, photogate, or the like.The support circuitry 30 generally includes one or more switchingdevices such as transistors that reset and facilitate read out of thepixel 26. One or more select signals provided to a select signal input32 (from the pixel selection circuitry 16) initiate reset and read outthe pixel 26. During a read operation, analog signals indicative of theamount of light detected by the pixel 26 are provided to a column busoutput 34, which is coupled to the sampling circuitry 18.

For purposes of discussion herein, the pixel 26 generally operates inone of three states: idle, reset, and read out. In an idle state,photons that collide with the light detecting element 28 dislodgeelectrons that accumulate in a potential well of the light detectingelement 28. The number of electrons that accumulate in the potentialwell of the light detecting element 28 is proportional to the number ofphotons that contact the light detecting element 28. In the idle state,the components of the support circuitry 30 remain off. Accordingly, thepixel 26 consumes minimal if any power and dissipates little if any heatin the idle state. The idle state is also referred to as an inactivestate herein. During a reset operation, one or more reset switchingcomponents (e.g., transistors) in the support circuitry 30 flush out theelectrons accumulated in the potential well of the light detectingelement 28. Some power is consumed by the one or more reset switchingcomponents and thus some heat is generated by the pixel 26 during thereset operation. During a read operation, one or more read out switchingelements in the support circuitry are turned on to process and transferthe charge stored in the potential well of the light detecting element28 (e.g., as a voltage or current) to the column bus output 34. Somepower is consumed by the one or more read out switching components andthus some heat is generated by the pixel 26 during the read operation.

Conventionally, all of the pixels in the active pixel array 12 are readout to provide a single image frame from the image sensor 10. Generally,this is done as part of a rolling shutter readout, wherein every pixelin a row of pixels is reset, allowed to remain in an idle state for someamount of time (i.e., the integration time), then read out. This processrepeats for each row of pixels until all of the pixels in the activepixel array 12 have been reset and subsequently read out. As the numberof rows in an active pixel array 12 increases, the time to capture andread out a single image frame also increases. This may limit the numberof image frames that can be captured in a given period of time, known asthe frame rate of the image sensor. A limited frame rate may beproblematic in some applications. Additionally, the resulting digitizedimage frame including pixel data for all of the pixels in the activepixel array 12 may be quite large. This may result in increased transfertime of the digitized image frame between the image sensor 10 andexternal processing circuitry (not shown), as such a transfer is oftenperformed serially. Further, this may result in increased analysis timeof the digitized image frame by said external processing circuitry, forexample, to detect occupancy in the image frame or a set of imageframes. Finally, as discussed above, every reset and read out of a pixelin the active pixel array 12 consumes power and dissipates heat. Overtime, continually resetting and reading out every pixel in the activepixel array 12 may raise the temperature of the active pixel array 12.As the temperature of the active pixel array 12 increases, the signal tonoise ratio of each one of the pixels therein decreases due to thermalnoise. This may make it difficult to analyze the resulting image frameor set of image frames, for example, to detect occupancy.

The inventors of the present disclosure discovered that it is highlyinefficient and unnecessary to analyze the entirety of an image frame orset of image frames to detect a transition from an unoccupied state toan occupied state. This is because persons entering the field of view ofan image sensor necessarily must first pass through one or more areaswithin the field of view before being present in other parts of thefield of view. For example, for an image sensor in the middle of a room,a person must necessarily pass through an outer edge of the field ofview before being present in the center of the field of view. As anotherexample, for an image sensor located in a hallway where the field ofview includes the entirety of the area between the two enclosing wallsof the hallway, a person must necessarily pass through either the top orthe bottom of the field of view before being present in the center ofthe field of view. As yet another example, for an image sensor with afield of view including the only door to a room and it is known that theroom is empty (e.g., due to the absence of occupancy for a given periodof time), a person must necessarily pass through the area of the fieldof view near the door before being present in any other part of thefield of view.

Accordingly, FIG. 3 is a flow diagram illustrating a method fordetecting occupancy using an image sensor according to one embodiment ofthe present disclosure. The method starts by obtaining pixel data for asubset of pixels in an active pixel array of an image sensor such thatthe pixels not in the subset remain inactive (step 100). As discussedherein, when a pixel is inactive or idle, the supporting circuitrytherein is off and thus the pixel is consuming minimal if any power andproducing minimal if any heat. Accordingly, obtaining the pixel datafrom the subset of pixels in the active pixel array involves reading outonly the subset of pixels while allowing the remaining pixels to remaininactive. Next, the pixel data is analyzed to determine if a person hasentered the field of view of the image sensor (step 102). Detailsregarding analyzing the pixel data to determine occupancy therefrom canbe found in FIGS. 9-12B below and U.S. patent application Ser. No.15/191,753, now U.S. Pat. No. 10,306,738; Ser. No. 15/887,096, nowpublished as U.S. Patent Application No. 2018/0225834; and Ser. No.15/681,941, now U.S. Pat. No. 10,165,650, the contents of which arehereby incorporated by reference in their entirety. It is thendetermined if a person has entered the field of view (step 104). If aperson has entered the field of view, additional pixel data mayoptionally be obtained (step 106), where the additional pixel datacontains pixel data for a larger portion of pixels in the active pixelarray than the subset of pixels. For example, the additional pixel datamay contain pixel data for all of the pixels in the active pixel array.Finally, the additional pixel data may be analyzed to determine if anarea within the field of view of the image sensor is occupied (step108). Step 108 may be used as a verification of step 102, or may be usedto verify the continuing occupancy of the area within the field of viewof the image sensor. Once again, details regarding analyzing the pixeldata to determine occupancy therefrom can be found in theabove-mentioned patent applications.

By obtaining pixel data for only the subset of pixels in the activepixel array such that the pixels not in the subset remain inactive, thetemperature of the active pixel array can be kept much lower than if allof the pixels in the active pixel array were read out. This results insignificant improvements in the signal to noise ratio of the pixelswithin the subset due to a reduction in thermal noise. Such improvementsmay be significantly evident in environments that are hot and dark,since signal to noise ratios in these environments tend to be highlyunfavorable. Further, analyzing the pixel data to determine if a personhas entered the field of view of the image sensor is far lesscomputationally expensive due to the reduction in the total amount ofpixel data to analyze. Optionally obtaining and analyzing the additionalpixel data may improve the reliability of detecting occupancy in thismanner.

By choosing the subset of pixels in the active pixel array wisely, theefficacy of detection of a person entering the field of view of theimage sensor can be very high. Notably, the subset of pixels may bechosen such that all of the pixels within the subset reside in a singlecontiguous area or such that the pixels within the subset are located inseparate, discrete areas. In embodiments in which the pixels within thesubset reside in a single contiguous area, such a contiguous area may bedefined by a polygon containing any number of sides, and at least fivesides (i.e., a non-rectangular shape) in some embodiments.

In one embodiment, the subset of pixels is chosen such that they residealong an outer border of the active pixel array as illustrated in FIG.4. Specifically, FIG. 4 illustrates an exemplary readout pattern 36 foran active pixel array in which only the pixels along the outer edges ofthe active pixel array (illustrated by a group of shaded pixels) areread out, while the remaining pixels along the interior of the activepixel array remain inactive or idle. A person entering the field of viewof an image sensor will necessarily first pass through an outer edge ofthe field of view before arriving in any other portion thereof.Accordingly, by analyzing pixel data from only the pixels in an areaalong the outer edges of the active pixel array, one can easily detectpersons entering the field of view of the camera using only a subset ofthe pixels therein.

In some embodiments, it may be desirable to choose the subset of pixelssuch that it resides around or near one or more ingress and/or egresspoints within the field of view of the image sensor. Accordingly, FIG. 5is a flow diagram illustrating a method for detecting occupancy using animage sensor according to an additional embodiment of the presentdisclosure. The method starts by determining a portion of a field ofview of an image sensor that is near an ingress and/or egress point(step 200). The ingress and/or egress point may be a door, a hallway, orthe like. In general, the ingress and/or egress point is one that aperson must travel through in order to gain access to the remainingportion of the field of view. Next, a subset of pixels in an activepixel array that detect light in the area near the ingress and/or egresspoint is determined (step 202). This may involve a simple mapping of anarea of the field of view to corresponding pixels in the active pixelarray that detect light within this area. Next, pixel data is obtainedfrom the subset of pixels in the active pixel array such that the pixelsnot in the subset remain inactive (step 204). As discussed herein, whena pixel is inactive, the supporting circuitry therein is off and thusthe pixel is consuming minimal if any power and producing minimal if anyheat. Accordingly, obtaining the pixel data from the subset of pixels inthe active pixel array involves reading out only the subset of pixelswhile allowing the remaining pixels to remain inactive. The pixel datais then analyzed to determine if a person has entered the field of viewof the image sensor (step 206). Once again, details regarding analyzingthe pixel data to determine occupancy therefrom can be found in theabove-mentioned patent applications. If a person has entered the fieldof view, additional pixel data may optionally be obtained (step 208),where the additional pixel data contains pixel data for a larger portionof pixels in the active pixel array than in the subset of pixels. Forexample, the additional pixel data may contain pixel data for all of thepixels in the active pixel array. Finally, the additional pixel data maybe analyzed to determine if an area within the field of view of theimage sensor is occupied (step 210). Step 210 may be used as averification of step 206, or may be used to verify the continuingoccupancy of the area within the field of view of the image sensor. Onceagain, details regarding analyzing the pixel data to determine occupancytherefrom can be found in the above-mentioned patent applications.

FIGS. 6A and 6B illustrate exemplary readout patterns 36 for an activepixel array according to various embodiments of the present disclosure.With respect to FIG. 6A, only those pixels in the lower left corner ofthe active pixel array (illustrated by a group of shaded pixels) areread out, while the remaining pixels are inactive or idle. Such apattern may be effective, for example, when there is only one ingressand/or egress point to the field of view of the image sensor (e.g., adoor leading to a room in which the image sensor is located) and it isin the lower left corner thereof. Notably, the subset of pixels forms apolygon including six sides. As discussed above, the subset of pixelsmay be chosen to occupy an arbitrary area defined by a polygon havingany number of sides.

With respect to FIG. 6B, only those pixels along a lower left edge,lower bottom edge, and lower right edge of the active pixel array(forming a “U” shape) are read out, while the remaining pixels areinactive or idle. Such a pattern may be effective, for example, when itis known that person must pass through the lower outside edges of theimage frame before being present in any other portion of the field ofview.

With respect to FIG. 6C, the subset of pixels includes a first region ofinterest 38A in the lower left corner of the active pixel array and asecond region of interest 38B on the right side of the active pixelarray. Notably, the first region of interest 38A and the second regionof interest 38B are not contiguous. While not shown, the first region ofinterest 38A and the second region of interest 38B may also be polygonshaving any number of sides. Further, while only the first region ofinterest 38A and the second region of interest 38B are shown, the subsetof pixels may include any number of separate regions of interest thatare either discrete or semi-contiguous without departing from theprinciples of the present disclosure. The pattern shown in FIG. 6C maybe effective, for example, when a person must pass through the lowerleft side of the field of view or the right side of the field of viewbefore being present in any other portion of the field of view.

Conventional image sensors are not able to read out pixels in an activepixel array in an arbitrary pattern. The image sensor 10 discussedherein may include modifications thereto such that the pixel selectioncircuitry 16 is capable of selecting pixels in an arbitrary fashion inorder to read out only a subset of the pixels in the active pixel array12 such that the subset includes polygons having any number of sidesand/or noncontiguous regions of interest.

The image sensor 10 may further be configured to read out the pixels inthe subset of pixels in a continuous fashion such that there are nopauses for pixels that are not being read out. This may require thecontrol circuitry 14 to compute and implement specialized timing for thevarious parts of the image sensor 10 specific to the subset of pixelssuch that the pixel data can be properly sampled. Accordingly, thecontrol circuitry 14 may be configured to alter the timing of pixelselection by the pixel selection circuitry 16 in order to provide aproper read out of the subset of pixels. Further, the control circuitry14 may be configured to change operating parameters of the samplingcircuitry 18 and the analog-to-digital converter circuitry in order toproperly digitize the pixel data from the subset of pixels. Continuouslyreading out the subset of pixels without pausing for those pixels thatare not being read out may significantly lower read times and thus allowfor increases in frame rate above and beyond that which is achievable bya conventional image sensor.

Finally, the control circuitry 14 may be configured to change operatingparameters of the output register 22 such that the pixel data for thesubset of pixels is properly arranged and thus communicated to externalcircuitry for analysis. In particular, the image sensor 10 may beconfigured to capture, store, and facilitate transfer of the pixel datafor the subset of pixels as a sparse data structure that does notinclude reserved spots (i.e., blank spaces) for pixels that are not inthe subset of pixels. This may allow for a much smaller data structureto be provided via the output 24, improving transmit times when providedto another device.

The image sensor 10 discussed herein may be incorporated into anintelligent lighting fixture 40 as shown in FIG. 7. The intelligentlighting fixture 40 includes the image sensor 10, driver circuitry 42,communications circuitry 44, and a solid-state light source 46. Thedriver circuitry 42 is coupled to the image sensor 10, thecommunications circuitry 44, and the solid-state light source 46. Thedriver circuitry 42 may use the communications circuitry 44 tocommunicate with other devices such as other lighting fixtures within adistributed lighting network. Further, the driver circuitry 42 maycontrol one or more light output parameters (e.g., brightness, colortemperature, color rendering index, and the like) of the solid-statelight source by providing one or more driver signals thereto. Finally,the driver circuitry 42 may obtain and analyze pixel data from the imagesensor according to the methods discussed above with respect to FIG. 3and FIG. 5 to determine and react to occupancy. By using pixel data foronly a subset of pixels in the active pixel array 12 of the image sensor10, the processing resources of the driver circuitry 42 may besignificantly conserved. This may in turn lead to reduced powerconsumption of the lighting fixture 40, reduced cost due to the reducedprocessing requirements of the driver circuitry 42, and improvedlongevity of the driver circuitry 42.

The intelligent lighting fixture 40 may be one of many intelligentlighting fixtures 40 in an intelligent lighting network 48, as shown inFIG. 8. The intelligent lighting fixtures 40 may communicate with oneanother in order to provide certain functionality such as responding tooccupancy events. Each one of the intelligent lighting fixtures 40 maybe configured to detect occupancy using the image sensor 10 as discussedabove. However, each one of the intelligent lighting fixtures 40 may beconfigured with a different read out pattern for the active pixel array12 of the image sensor 10, such that the subset of pixels used todetermine occupancy is different for different ones of the intelligentlighting fixtures 40.

In one embodiment, the intelligent lighting fixtures 40 may beconfigured to save processing resources by only requiring certain onesof the lighting fixtures 40 to detect occupancy when a space isunoccupied. In particular, those intelligent lighting fixtures 40 wherea field of view of the image sensor 10 thereof includes an ingressand/or egress point to a space in which the intelligent lightingfixtures 40 are located may be tasked with detecting occupancy when thespace is currently unoccupied, while the remaining intelligent lightingfixtures 40 are not required to do so. Those lighting fixtures 40 wherea field of view of the image sensor 10 thereof does not include aningress and/or egress point to the space do not need to detect occupancywhen the space is currently unoccupied, because a person cannot enterthe space without first passing through an ingress and/or egress pointthereto. When occupancy is detected by one of the intelligent lightingfixtures 40 tasked with detecting occupancy, the remaining intelligentlighting fixtures 40 may begin to detect occupancy as well in order toverify occupancy or more accurately or precisely determine occupancy.

Of those intelligent lighting fixtures 40 that are tasked with detectingoccupancy in the intelligent lighting network 48 when the space iscurrently unoccupied, they may utilize the methods discussed abovewherein only a subset of pixels in the active pixel array 12 of theimage sensor 10 are used for doing so. The read out patterns for eachone of the lighting fixtures 40 may be configured to accurately detectoccupancy with minimal processing overhead as discussed above. The readout patterns may be determined by persons familiar with the space andprogrammed into the intelligent lighting fixtures 40, or may bedetermined by the intelligent lighting fixtures 40 themselves or adevice in the intelligent lighting network 48 with access to sensor datafrom the intelligent lighting fixtures 40, for example, using learningalgorithms.

Turning now to FIG. 9, a flow diagram is provided to illustrate both thegeneral operation of each intelligent lighting fixture 40 as well astracking of occupants within a given environment. Each intelligentlighting fixture 40 will control its light output for generalillumination based on information or instructions provided by otherentities and/or sensors (step 300). For example, light output may becontrolled, such as being turned on, turned off, or dimmed to a desiredlevel, based on information received from one or any combination of anassociated wall controller, control node, system controller, processingnode, other lighting fixture, and the like.

In addition to providing light for general illumination, eachintelligent lighting fixture 40 is configured to determine the number ofoccupants in the associated field of view (step 302) and provideoccupancy information, based on the number of occupants in theassociated field of view, to a remote entity, such as the systemcontroller, control node, processing node, and the like (step 304). Inessence, the occupancy information for a given intelligent lightingfixture 40 generally corresponds to the number of occupants within thelighting fixture's field of view. Based on the occupancy information forthe intelligent lighting fixture 40 in a given area, the number ofoccupants for the given area may be calculated by summing the number ofoccupants that are in the fields of view for each of the lightingfixtures in the given area. In certain embodiments, steps are taken toavoid redundantly counting an occupant that resides in multiple fieldsof view at the same time. Details are provided further below.

As indicated above, controlling the light output (step 300), determiningthe number of occupants in an associated field of view (step 302), andproviding occupancy information to a remote entity (step 304), areprovided on a fixture-by-fixture basis. Each intelligent lightingfixture 40 uses the associated image sensor 10 to track occupants on aper occupant basis. As such, one or more occupants may be tracked by agiven intelligent lighting fixture 40 at any given time. In oneembodiment, the intelligent lighting fixture 40 will use its sensors todetect motion caused by a moving object in the associated field of view(step 306) and classify the object as either an occupant or non-occupant(step 308). An occupant is considered as a person (human), while anon-occupant is generally considered an object, or anything other than aperson. If an object is classified as an occupant, the occupant istracked while the occupant remains in the associated field of view (step310).

When the occupant moves or is predicted to move outside of theassociated field of view, the intelligent lighting fixture 40 willcoordinate with neighboring intelligent lighting fixtures 40 tofacilitate a handoff of the occupant tracking to the neighboringintelligent lighting fixture 40 that provides a field of view to whichthe occupant has moved or is predicted to move (step 312). Thedetection, classification, tracking, and handoff steps 306-312 mayprovide information that is helpful when both controlling the lightoutput (step 300) as well as determining the number of occupants in thefield of view of a given intelligent lighting fixture 40 (step 302). Inessence, as occupants are detected in or leave the associated field ofview, the intelligent lighting fixture 40 will dynamically update andreport on the total number of occupants in its associated fieldaccordingly. Again, occupants within the associated field of view of aparticular intelligent lighting fixture 40 may be tracked on anindividual basis, where the intelligent lighting fixture 40 may trackmultiple occupants at any given time.

FIG. 10 is a flow diagram illustrating how individual occupants aredetected and tracked within a given lighting fixture's field of viewusing the image sensor 10 and/or a passive infrared (PIR)-basedoccupancy sensor. The image sensor 10 is used for high-resolutiondetection, classification and tracking of occupants within a field ofview for the associated intelligent lighting fixture 40. The PIR-basedoccupancy sensor is used for low-resolution occupancy detection. Assuch, various other types of sensors, such as acoustic, thermal, image,and the like may be employed for a sensor. It is assumed that controlelectronics and associated software of the intelligent lighting fixture40 use information gathered from the image sensor 10 and/or additionalsensors, and perhaps neighboring intelligent lighting fixtures 40, toprovide the following functionality. Those skilled in the art willrecognize that such functionality may be integrated within ordistributed among various hardware and/or software components of one ormore intelligent lighting fixtures 40 and associated devices.

The process starts when the intelligent lighting fixture 40 analyzesinformation provided by one or both of the image sensor 10 and thePIR-based occupancy sensor to monitor for motion caused by the movementof an object within the lighting fixture's field of view or generalvicinity (step 400). If motion is not detected (step 402), theintelligent lighting fixture 40 will continue to monitor for motion(step 400). If motion is detected (step 402), the intelligent lightingfixture 40 will analyze information provided by the image sensor 10 inan effort to determine if the object is an occupant who has previouslybeen detected and is currently being tracked (step 404). Motiondetection employs the use of one or more sensors. As an example for animage sensor 10, the intelligent lighting fixture 40 will analyzecaptured image information to detect motion. For a PIR-based occupancysensor, the intelligent lighting fixture 40 will receive an outputindicative of motion within the sensor's field of view. The intelligentlighting fixture 40 may process the information from various sensors todetect or otherwise identify motion.

If the object is not an occupant who is currently being tracked, theintelligent lighting fixture 40 will analyze the information provided bythe image sensor 10 to classify the object as either an occupant or anon-occupant, where an occupant is a person and a non-occupant is aninanimate object (step 406). Those skilled in the art will recognizevarious schemes to classify an object. Examples include histogram oforiented gradients (HOG) schemes, which use machine learning to classifyobject appearance and shape. Incorporation of Principal ComponentsAnalysis (PCA) into the HOG schemes provide for particularly robust andreproducible classification of occupants and groups of occupants.HOG-based classification is viewed as sufficiently robust, yet notoverly computationally burdensome for a lighting application. If evengreater accuracy is required, the use of sparse convergent neuralnetworks (SCNN) may be employed; however, the use of SCNN may requiremore microprocessor memory and/or speed than HOG-based schemes. The SCNNscheme seeks to reduce the complexity of established neural networkschemes, thereby reducing response time and computation resources evenin visually complex environments.

If the object is a non-occupant (step 408), the intelligent lightingfixture 40 will continue to monitor for motion (step 400). If the objectis an occupant (step 408), the intelligent lighting fixture 40 willanalyze the information provided by the image sensor 10 to track theoccupant within the field of view associated with the image sensor 10(step 410). The function of tracking may range from simply determiningthat the occupant is within the field of view to determining one or moreof a precise location within the field of view, a direction of travel,and a velocity of travel, wherein the direction and velocity of travelmay be represented with an appropriate vector that is associated with adirection and magnitude, which corresponds to velocity.

The intelligent lighting fixture 40 may also analyze the informationreceived from the image sensor 10 to identify one or more physicalcharacteristics associated with the occupant, such as shape, size,colors, patterns, and the like. These characteristics are helpful forre-identifying an occupant when tracking is lost within the lightingfixture's field of view, recognizing that the occupant is a singleoccupant when the occupant resides in an area where the fields of viewfrom adjacent intelligent lighting fixtures 40 overlap, and handing offtracking of an occupant from one intelligent lighting fixture 40 toanother when the occupant moves from one lighting fixture's field ofview and to another.

The tracking function may employ various tracking schemes, includingKalman filtering, which provides a streamlined and effective techniquefor persistently tracking objects.

When motion is detected and the object detected is a previously trackedoccupant (step 404), the intelligent lighting fixture 40 may skip thesteps of classifying the object (steps 406 and 408), since the object isalready known as an occupant, and move directly to tracking the occupant(step 410). As indicated above, this situation may occur when anoccupant has not left the field of view for the intelligent lightingfixture 40, but tracking is lost for some reason. As the occupant moveswithin the field of view, the intelligent lighting fixture 40 willdetect motion associated with the occupant moving (step 402), analyzeinformation provided by the image sensor 10, and recognize that theobject is a previously tracked occupant based on the analysis of theinformation provided by the image sensor 10 (step 404).

While tracking an occupant, the intelligent lighting fixture 40 may beconfigured to detect when tracking is lost for the occupant (step 412)as well as detect or predict that the occupant is leaving the field ofview (step 414) for the intelligent lighting fixture 40. Assumingtracking for the occupant is not lost (step 412) in that the occupanthas not left or is not leaving the field of view for the intelligentlighting fixture 40, tracking will continue (step 410). If tracking forthe occupant is lost (step 412), the intelligent lighting fixture 40 maybe configured to check information from a second sensor, which in thisembodiment is PIR-based occupancy sensor, and determine whether or notoccupancy is being detected via the second sensor (step 416). If theinformation from the second sensor indicates that occupancy is stillbeing detected (step 418), the intelligent lighting fixture 40 willcontinue trying to track the occupant (step 410).

If the information from the second sensor indicates that occupancy isnot detected (step 418), the intelligent lighting fixture 40 willcommunicate with adjacent intelligent lighting fixtures 40 that provideneighboring fields of view to that provided by the intelligent lightingfixture 40 (step 420). The interaction between neighboring lightingfixtures may take many forms. For example, the intelligent lightingfixture 40 may ask its neighbors if any occupants have recently appearedin their fields of view from the field of view of the intelligentlighting fixture 40. The intelligent lighting fixture 40 may alsoprovide its neighbors with recent information bearing on one or more ofrecent location, direction, velocity, and physical characteristics ofthe lost occupant and the neighbors will compare the informationprovided by the intelligent lighting fixture 40 with any occupantscurrently being tracked by the neighbors.

If a determination is made that the lost occupant is not in theneighbor's field of view (step 422), the intelligent lighting fixture 40reverts back to monitoring for motion (step 400). At this point, theintelligent lighting fixture 40 is able to recognize the lost occupantif the occupant is once again detected in the lighting fixture's fieldof view. Notably, the intelligent lighting fixture 40 is alwaysmonitoring the field of view for new objects that could be occupantswhen the process repeats.

If a determination is made that the lost occupant is in the neighbor'sfield of view (step 422), the intelligent lighting fixture 40 willprovide any handoff information necessary for handing off tracking ofthe lost occupant to the neighbor that has picked up the occupant in itsfield of view (step 424) and then facilitate handoff of the occupant tothe neighbor (step 426). The handoff information may include acombination of location, direction, velocity, and physicalcharacteristics of the lost occupant. This list is not inclusive, andthose skilled in the art will recognize other pertinent information thatmay be helpful in various embodiments. Kalman filtering or the like maybe used to facilitate handoffs.

Returning to step 414, another trigger for handing off tracking of anoccupant to the neighbor is when the intelligent lighting fixture 40 isactively tracking the occupant (step 410) and predicts, or determines,that the occupant is leaving the lighting fixture's field of view (step414). If the intelligent lighting fixture 40 can identify the neighbortoward which the occupant is moving, the intelligent lighting fixture 40will prepare the handoff information for the occupant (step 424) andcommunicate with the neighbor to share the handoff information andfacilitate the handoff (step 426). If the intelligent lighting fixture40 cannot identify the neighbor toward which the occupant is moving, theintelligent lighting fixture 40 will prepare the handoff information forthe occupant and communicate with other intelligent lighting fixtures 40in the lighting network N1 to look for an occupant entering their fieldsof view. A neighbor receiving the occupant may acknowledge receipt ofthe occupant and increase its occupancy count to account for the newoccupant in its field of view. The intelligent lighting fixture 40 willreduce its occupancy count to account for having an occupant leave itsfield of view.

Step 428 indicates that a handoff of an occupant from a neighbor mayenter the process at the tracking phase (step 410); however, otherembodiments may bring in an occupant that is being handed off from aneighbor at any other point in the process.

FIGS. 11A through 11F illustrate imagery of a single field of view (FOV)derived from an image sensor 10 at six points in time (t1-t6). In thisexample, two occupants, person P1 and person P2, as well as onenon-occupant object N, are present at one time or another in the fieldof view. Throughout the progression, person P1 enters the field of viewfrom the left, progresses from left to right through the field of view,and exits the field of view on the right. Using information collectedfrom the image sensor 10, the intelligent lighting fixture 40 employs afirst process to detect and determine that the person P1 is an occupantfor occupancy purposes as well as track person P1 as she enters,progresses through, and exits the field of view.

Using the same information collected from the image sensor 10, theintelligent lighting fixture 40 employs a second process to detect themovement of the non-occupant object N, once the non-occupant object Nbegins moving at time t3 (FIG. 11C). Based on the collected information,the intelligent lighting fixture 40 will be able to determine that thenon-occupant object N is not an occupant for occupancy purposes. Usinginformation collected from the image sensor 10, the intelligent lightingfixture 40 employs a third process to detect and determine that theperson P2 is an occupant for occupancy purposes as well as track personP2 as he enters and diagonally progresses through the field of view.

The intelligent lighting fixture 40 will update its occupancy countaccordingly as persons P1 and P2 enter and leave the field of view(FOV). The presence or movement of the non-occupant object N will notaffect the occupancy count. The intelligent lighting fixture 40 willreport the occupancy information periodically or as it changes to aremote entity, such as a system controller, control node, processingnode, or the like, which may take appropriate action based on theoccupancy information or a change in the occupancy information.

When there are overlapping fields of view provided by the variousintelligent lighting fixtures 40, logic trees may be developed torecognize and act on movement of occupants through the borders(periphery) of the various fields of view with neighboring intelligentlighting fixtures 40 being notified of imminent occupant arrival basedon their position relative to the reporting intelligent lighting fixture40. In one embodiment, handoff logic is developed to effectively handlethe case where an occupant resides in an overlapping portion of twofields of view by incorporating probabilistic weighting to define whichintelligent lighting fixture 40 “owns,” and thus should account for, theoccupant.

When determining overall occupancy for a particular space, identifyingthe intelligent lighting fixtures 40 that are within the space as wellas the relative location of the intelligent lighting fixtures 40 inspace is very helpful in maintaining accurate occupancy accounts for thespace. FIG. 12A illustrates an office environment that has a conferenceroom, two offices, and a cubicle area having four cubicles. Each dashedsquare represents the field of view of an overhead intelligent lightingfixture 40. As illustrated, many of the fields of view of the lightingfixtures can overlap with one another.

As graphically presented in FIG. 12A, the rooms in which the intelligentlighting fixtures 40 are located as well as the relationships, orrelative locations, of intelligent lighting fixtures 40 within theoffice environment are clearly depicted. However, when the intelligentlighting fixtures 40 are installed, neither the intelligent lightingfixtures 40 nor any remote entities that may control or receiveinformation from them are able to discern location or relationshipinformation without automated or manual mapping of some form. FIG. 12Bgraphically represents the fact that when initially installed,intelligent lighting fixtures 40 are essentially randomly located andoriented. During a commissioning process, a user may employ a remoteentity, such as a control node 36, to access the fields of view for thevarious intelligent lighting fixtures 40 and map them according to theirorientation and location throughout the office environment. The user mayalso assign the intelligent lighting fixtures 40 to occupancy groups,based on the particular space in which the intelligent lighting fixtures40 reside. For example, the conference room occupancy group will havesix intelligent lighting fixtures 40, the occupancy group for each ofthe offices will include two intelligent lighting fixtures 40, and theoccupancy group for the cubicle area will include six intelligentlighting fixtures 40.

Depending on the capabilities of the system, the user may be able toprecisely map and orient intelligent lighting fixtures 40, such thateach intelligent lighting fixture 40 may be provided with sufficientinformation to identify the other intelligent lighting fixtures 40 thatare within the particular occupancy group of the intelligent lightingfixture 40, those neighboring intelligent lighting fixtures 40 that areimmediately adjacent the intelligent lighting fixture 40, the role ofthe position of the neighboring intelligent lighting fixtures 40relative to the intelligent lighting fixture 40, and the like. As such,each intelligent lighting fixture 40 may be provided with informationidentifying the occupancy group within which it resides, the neighboringintelligent lighting fixtures 40 that are immediately adjacent otherintelligent lighting fixtures 40, and a relative location of theneighboring intelligent lighting fixtures 40. This information isparticularly helpful for the tracking and handoff functions, which weredescribed above.

The mapping process may be automated to varying degrees. In a highlyautomated embodiment, a processing node or the like will collect imageinformation from the various intelligent lighting fixtures 40, analyzethe content and/or characteristics of the image information, and createa map of the intelligent lighting fixtures 40 as described above andgraphically represented in FIG. 12A. During such a process, the imageanalysis may identify objects, patterns, colors, light intensities,lighting gradients, and the like in an effort to piece the fields ofview into a cohesive map. In this particular example, portions of theperimeter of the fields of view that are likely to include helpfulalignment cues, such as walls, bisected objects, and the like, and arehighlighted in bold. These portions of the perimeter the fields of viewinclude walls, objects that span fields of view, and the like.

Those skilled in the art will recognize improvements and modificationsto the preferred embodiments of the present disclosure. All suchimprovements and modifications are considered within the scope of theconcepts disclosed herein and the claims that follow.

What is claimed is:
 1. An intelligent lighting fixture comprising: asolid-state light source; an image sensor comprising: an active pixelarray comprising a plurality of pixels; and image sensor controlcircuitry configured to: operate in a first mode of operation whereinthe image sensor control circuitry is configured to perform readoperations only on a subset of the plurality of pixels such that theplurality of pixels that are not in the subset of the plurality ofpixels remain inactive; and operate in a second mode of operationwherein the image sensor control circuitry is configured to perform aread operation on all of the plurality of pixels; and driver circuitryconfigured to: control one or more light output parameters of thesolid-state light source; when the image senor control circuitryoperates in the first mode of operation, obtain pixel data for thesubset of the plurality of pixels from the image sensor; analyze thepixel data to classify an object detected by the image sensor as anoccupant or a non-occupant; and in response to classifying the object asan occupant, cause the image sensor control circuitry to operate in thesecond mode of operation.
 2. The intelligent lighting fixture of claim 1wherein the subset of the plurality of pixels comprises a first regionof interest and a second region of interest that is noncontiguous withthe first region of interest.
 3. The intelligent lighting fixture ofclaim 1 wherein the driver circuitry is further configured to analyzethe pixel data to track the object within an environment captured by theimage sensor.
 4. The intelligent lighting fixture of claim 3 wherein thedriver circuitry is further configured to communicate with a neighboringintelligent lighting fixture in response to classifying the object asthe occupant.
 5. The intelligent lighting fixture of claim 4 wherein thedriver circuitry is further configured to handoff tracking to theneighboring intelligent lighting fixture in response to determining theoccupant has entered a field of view of the neighboring intelligentlighting fixture.
 6. The intelligent lighting fixture of claim 1 whereinthe subset of the plurality of pixels is defined by a polygon having atleast 5 sides.
 7. The intelligent lighting fixture of claim 6 whereinthe driver circuitry is further configured to analyze the pixel data totrack the occupant within an environment captured by the image sensor.8. The intelligent lighting fixture of claim 1 wherein the subset of theplurality of pixels is defined by a rectangular area along one or moreoutside edges of the active pixel array.
 9. The intelligent lightingfixture of claim 8 wherein the subset of the plurality of pixels forms aframe around the one or more outside edges of the active pixel arraysuch that the plurality of pixels that are not in the subset of theplurality of pixels form a rectangle that is inset within the subset ofthe plurality of pixels.
 10. The intelligent lighting fixture of claim 1wherein each one of the plurality of pixels comprises: a light detectingelement configured to transform light into an analog signal; andsupporting circuitry configured to: during a read operation, process theanalog signal from the light detecting element to provide a processedanalog signal and provide the processed analog signal to downstreamcircuitry in the image sensor; and remain inactive when a read operationis not occurring.
 11. The intelligent lighting fixture of claim 1wherein each one of the plurality of pixels generates more heat during aread operation than when a read operation is not occurring.
 12. Theintelligent lighting fixture of claim 1 wherein each one of theplurality of pixels consumes more power during a read operation thanwhen a read operation is not occurring.
 13. The intelligent lightingfixture of claim 1 wherein noise within the subset of the plurality ofpixels is lower in the first mode of operation than in the second modeof operation.
 14. The intelligent lighting fixture of claim 1 whereinthe image sensor control circuitry is configured to capture and storepixel data from the subset of the plurality of pixels in a sparse datastructure such that only the pixel data from the subset of the pluralityof pixels is included in the sparse data structure.
 15. The intelligentlighting fixture of claim 14 wherein the image sensor control circuitryis configured to facilitate a transfer of the sparse data structure to aremote device.
 16. The intelligent lighting fixture of claim 1 whereinthe image sensor control circuitry is further configured to: in thefirst mode of operation, analyze the pixel data to determine if a personhas entered a field of view of the image sensor; and transition to thesecond mode of operation when the person has entered the field of viewof the image sensor.
 17. A method for detecting occupancy by anintelligent lighting fixture comprising an image sensor, the methodcomprising: obtaining pixel data from the image sensor by performingread operations only on a subset of pixels in an active pixel array ofthe image sensor such that pixels in the active pixel array that are notin the subset of pixels remain inactive; and analyzing the pixel data toclassify an object detected by the image sensor as an occupant or anon-occupant; and in response to classifying the object as an occupant,obtaining additional pixel data from the image sensor by performing aread operation on all pixels in the active pixel array of the imagesensor.
 18. The method of claim 17 wherein each one of the pixelsgenerates more heat during a read operation than when a read operationis not occurring.
 19. The method of claim 17 wherein each one of thepixels consumes more power during a read operation than when a readoperation is not occurring.
 20. The method of claim 17 wherein thesubset of pixels is defined by a rectangular area along one or moreoutside edges of the active pixel array.
 21. The method of claim 20wherein the subset of pixels in the active pixel array forms a framearound the one or more outside edges of the active pixel array such thatthe pixels in the active pixel array that are not in the subset ofpixels form a rectangle that is inset within the subset of pixels. 22.The method of claim 17 wherein: a field of view of the image sensorincludes one of an ingress point and an egress point to a space in whichthe image sensor is located; and the subset of pixels is located in theactive pixel array such that a person entering or leaving the space viaone of the ingress point and egress point will be detected by the subsetof pixels.
 23. The method of claim 17 further comprising analyzing theadditional pixel data to verify that an area within a field of view ofthe image sensor is occupied.
 24. The method of claim 17 furthercomprising analyzing the additional pixel data to determine if an areawithin a field of view of the image sensor remains occupied.